The present invention relates to an image processing apparatus and an image processing method, and a program, and particularly to an image processing apparatus and an image processing method, and a program that are suitable for use in demosaic processing (color interpolation processing or synchronization processing) for obtaining a color image signal from a mosaic image signal obtained by using a color solid-state image pickup element by interpolating a number of colors in all pixels.
A solid-state image pickup element such as a CCD image sensor, a CMOS image sensor, or the like, generally has a structure in which light receiving elements are arranged in the form of a lattice and an amount of charge produced by photoelectric conversion of each of the light receiving elements can be read sequentially. In general, these light receiving elements have a single spectral characteristic, and, therefore, an image signal obtained from the solid-state image pickup element is one channel (monochrome) with respect to color. Thus, when a color image (an image of three channels such as RGB, for example) is desired to be obtained with a single solid-state image pickup element, a solid-state image pickup element having a filter with a different spectral characteristic (color) for each light receiving element is used. An image pickup device using such a single color solid-state image pickup element is generally referred to as a single chip color image pickup device.
Since an image signal obtained from the color solid-state image pickup element represents a one-channel image, each pixel provides only an intensity of a color of a filter of a corresponding light receiving element. That is, the image signal obtained from the color solid-state image pickup element represents a mosaic image in terms of color. In order to obtain a multi-channel image from the mosaic image as an output image of the color solid-state image pickup element of the single chip color image pickup device, it is necessary to interpolate color information of pixels of the mosaic image in pixel positions around the pixels by appropriate image processing. Such image processing is generally referred to as color interpolation processing, demosaic processing, synchronization processing, or the like. Thus, the demosaic processing is essential in single chip color image pickup devices using color solid-state image pickup elements. Conventionally, various techniques for the demosaic processing have been developed.
A problem in this demosaic processing is characteristics of reproduction of a high-frequency image signal. In the solid-state image pickup element, different on-chip color filters are stuck on light receiving elements arranged in the form of a two-dimensional lattice and, therefore, pixels having the same color (that is, the same spectral sensitivity) are arranged at a pitch larger than an original arranging pitch of the light receiving elements. Thus color solid-state image pickup element is more prone to aliasing of the picked-up image than a monochrome solid-state image pickup element without on-chip color filters. When aliasing occurs in a luminance component of an image, it is often observed as jagginess at a contour portion of a subject. Suppression of jagginess is an important problem in the demosaic processing.
FIG. 1 shows a Bayer arrangement of a primary color system. The color arrangement now is most widely used. The Bayer arrangement uses color filters of three colors R, G, and B with G arranged in a checkered manner and with R and B arranged on a line-sequential basis (the Bayer arrangement of the primary color system hereinafter will be referred to simply as a Bayer arrangement). In the Bayer arrangement, since G is arranged in the checkered manner, a G signal is present in all horizontal and vertical phases, while since R and B are arranged on the line-sequential basis, a signal corresponding to each of R and B is present only in every other line in both a horizontal direction and a vertical direction.
Of R, G, and B in the Bayer arrangement, G, which has a spectral characteristic closest to a characteristic of visual sensitivity of a human, is most densely present. According to conventionally used techniques, a method of creating a luminance component from only G pixel information is mainstream. In this method, a limit of frequency of a luminance component reproduced, which limit is determined by G sampling frequency of the Bayer arrangement, is 0.5 cycle/pixel in the horizontal direction and the vertical direction, and 1/(2×sqrt(2)) cycle/pixel in an oblique 45-degree direction. While the limit is highest in the horizontal direction or the vertical direction in theory, G is present only in every other line in the horizontal direction or the vertical direction in practice. Therefore, in order to reproduce horizontal or vertical waves of 0.5 cycle/pixel, an optimum interpolation filter needs to be applied to each of the horizontal or vertical waves. In order to deal with this problem, there are techniques in which an optimum interpolation filter is provided for waves in the horizontal direction and the vertical direction, a wave direction in each local area on an image is determined, and a value obtained by synthesizing a result of the horizontal direction interpolation filter and a result of the vertical direction interpolation filter according to a result of the determination is set as an interpolated value (for example, Patent Document 1 and Non-Patent Document 1). By using these techniques, it is possible to reproduce high frequencies close to the theoretical limit of 0.5 cycle/pixel in the horizontal direction and the vertical direction using the G pixel information of the Bayer arrangement.
[Patent Document 1]
Japanese Patent Laid-open No. Hei 7-236147
[Non-Patent Document 1]
Murata, Mori, Maenaka, Okada, and Chihara, “Correlative Directional Interpolation for a Progressive Scan CCD,” The Journal of the Institute of Image Information and Television Engineers, Vol. 55, No. 1, pp. 120-132, 2001
On the other hand, it is theoretically impossible to make the spectral characteristics of the on-chip color filters used in the color solid-state image pickup element coincide with color matching functions of human vision. Accordingly, there is, for example, a method that uses an image pickup device for obtaining intensities of four colors provided by adding another color to the three primary colors of the Bayer arrangement, and obtains intensity values of the three primary colors by applying a linear matrix to the intensities of the four colors obtained by such an image pickup device, thus improving color reproducibility of the image pickup device. There is, for example, a technique that can improve color reproducibility using a color solid-state image pickup element having a four-color arrangement obtained by adding a yellow on-chip filter to the three conventional primary colors to obtain intensities of the four colors (for example Patent Document 2).
[Patent Document 2]
Japanese Patent Laid-open No. 2002-271804
However, it is not specifically shown in the conventional technique using color filters in the four-color arrangement what kind of demosaic processing is to be performed to obtain four color intensities synchronized in each pixel from the four-color arrangement.
A four-color arrangement of a complementary color system (cyan, magenta, yellow, and green) is conventionally used in image pickup elements such as video cameras and the like. Color filters having the four-color arrangement of the primary color system, are similarly in practical use. Since the conventional four-color arrangement of the primary color system makes it impossible to assign G to half of all pixels as in the Bayer arrangement, the conventional technique for reproducing high frequencies of a luminance component using the checkered G arrangement cannot be used. Thus, the use of the conventional four-color arrangement of the primary color system improves color reproducibility, but results in noticeable jagginess appearing without a high-frequency luminance component being reproduced or results in a smoothed dull image.